System and method for automatically validating agent implementation of training material

ABSTRACT

A system and method for testing an agent by a validation bot may include sending training material to an agent; automatically initiating, by the validation bot, a session with an agent by providing a phrase to the agent via a client media interface; obtaining a reply from the agent; mapping the reply of the agent to an agent intent; providing a response to the agent based on the agent intent and according to a predetermined session flow, wherein the predetermined session flow is based on the training material; repeating obtaining responses, mapping the responses and providing responses until a termination criterion is met; calculating a score of the agent according to scoring rules; and providing the score of the agent to a user.

PRIOR APPLICATION DATA

The present application is a continuation of prior U.S. patentapplication Ser. No. 16/855,106 filed 22 Apr. 2020, and entitled “SYSTEMAND METHOD FOR AUTOMATICALLY VALIDATING AGENT IMPLEMENTATION OF TRAININGMATERIAL”, which in turn is a continuation of prior U.S. patentapplication Ser. No. 15/938,586 filed Mar. 28, 2018, and entitled“SYSTEM AND METHOD FOR AUTOMATICALLY VALIDATING AGENT IMPLEMENTATION OFTRAINING MATERIAL”, each of which being incorporated by reference in itsentirety.

FIELD OF THE INVENTION

Embodiments of the invention relate to systems and methods forautomatically validating comprehension and implementation of trainingmaterial by agents at call centers. In particular, embodiments of theinvention relate to automatically validating agents based on trainingmaterial and agent actions.

BACKGROUND OF THE INVENTION

Company call centers or Internet-based support forums, also referred toherein as contact centers, put customers in contact with company agents(e.g. people working for the company providing customer service or otherinformation to customers) for assistance. Agents' performance may beimproved by increasing the agents' training. Agents may be trained forexample, using coaching packages including training material which maybe automatically distributed to agents, or in other manners. Typically,when a coaching package is launched or executed, agents are evaluatedshort time after the training material was sent to the agents, to testthat the content of the coaching package is clear and after apredetermined time, e.g., after few weeks or months to see if thecontent is memorable. However, validating and testing that agents areadhering to the training material typically involves evaluating a realinteraction with a real customer or with a human supervisorimpersonating customer. This imposes a challenge on call centersoperators, since in both cases highly trained supervisors are requiredfor listening to calls with customers or for conducting validationssessions. The burden is significant also due to the need to constantlytrain new agents due to low job retention time.

SUMMARY

According to embodiments of the invention a system and method fortesting an agent by a validation bot may include for example:

-   -   a. sending training material to an agent;    -   b. automatically initiating, by the validation bot, a session        with an agent by providing a phrase to the agent via a client        media interface;    -   c. obtaining a reply from the agent;    -   d. mapping the reply of the agent to an agent intent;    -   e. providing a response to the agent based on the agent intent        and according to a predetermined session flow, wherein the        predetermined session flow is based on the training material;    -   f. repeating operations c-e until a termination criterion is        met;    -   g. calculating a score of the agent according to scoring rules;        and    -   h. providing the score of the agent to a user.

According to embodiments of the invention the predetermined session flowmay include possible agent intents, possible responses to the agent,scoring rules, response rules, and state variables, wherein a state ofthe validation bot may be updated based on the agent intent, and whereinproviding the response to the agent may be performed based on the stateof the validation bot and on the bot response rules.

Embodiments of the invention may include providing a confidence scorefor the intent.

Embodiments of the invention may include repeating operations a-h for aplurality of agents.

Embodiments of the invention may include determining timing ofinitiating sessions with agents that need to be validated based on anumber of the agents that need to be validated and a time available forvalidating the agents, wherein timing may be determined so that thesessions are evenly distributed along the time available for validatingthe agents.

Embodiments of the invention may include scheduling the sessions fortimes in which there are more agents than an expected number requiredfor answering human clients.

Embodiments of the invention may include scheduling the sessions fortimes in which a difference between an expected number of agent requiredfor answering human clients and an actual number of agents is above apredetermined threshold.

According to embodiments of the invention the client media interface maybe one of chat box, short message service (SMS), voice and anycombination thereof.

Embodiments of the invention may include determining that the agent haspassed a verification test if the score of the agent exceeds apredetermined threshold and determining that the agent has failed theverification test if the score does not exceed the threshold; andproviding a pass/fail indication to the user.

Embodiments of the invention may include providing a recommendation torefine training material if a percentage of agents that have passed theverification test is below a predetermined threshold.

Embodiments of the invention may include, if the score is below apredetermined threshold, perform one of: sending training material tothe agent, providing pop-ups with correct answers to the agent inreal-time, and informing the agent of the low score.

According to embodiments of the invention, initiating the session may bedone using queues of an automated call distribution (ACD) system.

According to embodiments of the invention a system and method forsimulating a human client for testing agents may include for example:

-   -   a. initiating, by an automatic validation process, an        interaction with an agent by providing a phrase to the agent;    -   b. receiving an agent reply from the agent;    -   c. correlating the agent reply with an agent intent using a        natural language understanding engine;    -   d. updating a state of the validation process based on the agent        intent;    -   e. providing a response to the agent based on the agent intent        and on the state of the validation process;    -   f. repeating operations d-f until it is determined that the        process should stop;    -   g. calculating a rating of the agent according to scoring rules;        and    -   h. providing the rating of the agent to a user.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features, and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanying drawings in which:

FIG. 1 schematically illustrates a system for automatically testingcoaching effectiveness, according to embodiments of the invention;

FIG. 2 schematically illustrates a top-level design of a validation bot,according to embodiments of the invention;

FIG. 3 is a high-level flowchart of a method for automatically testingcoaching effectiveness by a validation bot, according to embodiments ofthe invention;

FIG. 4 is a high-level flowchart of the operations that may be performedin case of a failed test, according to embodiments of the invention;

FIG. 5 schematically illustrates a partial operational flow ofvalidation bot backend application, according to embodiments of theinvention;

FIG. 6 is a high-level flowchart of a method for evenly distributingsessions, according to embodiments of the invention;

FIG. 7 is a high-level flowchart of a method for distributing sessionsaccording to agent's availability, according to embodiments of theinvention; and

FIG. 8 is a high-level block diagram of an exemplary computing deviceaccording to some embodiments of the present invention.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION

In the following description, various aspects of the present inventionwill be described. For purposes of explanation, specific configurationsand details are set forth in order to provide a thorough understandingof the present invention. However, it will also be apparent to oneskilled in the art that the present invention may be practiced withoutthe specific details presented herein. Furthermore, well-known featuresmay be omitted or simplified in order not to obscure the presentinvention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing,” “computing,”“calculating,” “determining,” or the like, refer to the action and/orprocesses of a computer or computing system, or similar electroniccomputing device, that manipulates and/or transforms data represented asphysical, such as electronic, quantities within the computing system'sregisters and/or memories into other data similarly represented asphysical quantities within the computing system's memories, registers orother such information storage, transmission or display devices.

Customers may receive inconsistent service at telephone and Internethelp centers due to the variation in agent competency and training.Also, guidelines for agents may be occasionally updated. To improveperformance and keep agents updated, embodiments of the invention mayautomatically send training material to agents and test the coachingeffectiveness at selected time periods, e.g., close to launching, toverify that the content of the training material is clear, or after fewweeks or months to see if the content is memorable. Testing of thecoaching effectiveness may be performed automatically by a validationbot. Training material may include documents, slideshows, audio or videorecordings etc., that may include instructions, guidelines informationand exercises that may be sent to an agent for training purposes.Current automatic training programs do not integrate training andautomatic testing of the coaching effectiveness. Embodiments of theinvention may automate the training and evaluating of agents.Embodiments of the invention may provide a score to each agent and mayprovide valuable input to a system operator regarding overallperformance of agents. This input may be later used to improve coachingmaterial. Additionally, embodiments of the invention may enablescheduling validation sessions in a way that may minimize the negativeimpact of validation sessions on agents' availability. This may beachieved by integrating scheduling of validations sessions with aworkforce management program and scheduling sessions for hours in whichthe actual workforce is larger than a needed workforce, or bydistributing validation calls evenly over the working hours.

Embodiments of the invention pertain to the technology of call centersand may provide an improvement to this technology by for exampleimproving automated testing of agents and by improving automated testingand coaching effectiveness. In some embodiments training or coachingmaterial may be automatically sent to agents according to a trainingprogram. A testing session with an improved validation bot according toembodiments of the invention may be automatically initiated thereafterto assess the coaching effectiveness. Embodiments of the invention mayprovide feedback to a user, e.g., the agent or a supervisor, regardingthe performance of a single agent or a group of agents. Embodiments ofthe invention may improve the testing of agents by mapping the repliesof the agent to an agent intent and providing a response based on theagent intent and a session flow, wherein the predetermined session flowis based on the training material.

Furthermore, embodiments of the invention may improve the technologybehind the operation of call centers by scheduling bot validationsessions in hours in which the actual workforce is larger than a neededworkforce. For example, this may be achieved by using informationobtained from a workforce management program or application.

Embodiments of the invention may differ from a manual validation sincethe validation bot may employ a computerized language understandingengine for categorizing agent responses, thus enabling the validationbot to determine how to proceed in the validation session in accordanceto the bot flow. Additionally, the automatic scheduling may be differentfrom manual scheduling of sessions by offering integration withworkforce management program.

Reference is made to FIG. 1, which schematically illustrates a system100 for automatically testing coaching effectiveness, according toembodiments of the invention. System 100 may include support center 124to connect user devices 102 or validation bot 110 to agent devices 120.Contact center, also referred to as support center 124 may be or mayinclude company call centers for telephone-based support or onlinesupport forums for Internet-based support, for example, to providecustomers with technical support, sell products, schedule appointments,or otherwise interact or communicate with customers. Support center 124may also be connected to validation bot 110. Support center 124 mayinclude a switching station to order and connect each of a plurality ofuser devices 102 or validation bot 110 to one of a plurality of agentdevices 120 at the same time. Support center 124 may order customers andvalidation bot 110 in a queue, for example, to be served on a first-comefirst-serve basis. Support center 124 may connect each customer's userdevices 102 to the next available agent's device 120 from a pool ofavailable agents, for example, to provide the fastest agent responsetime. Support center 124 may connect validation bot 110 to selectedagents according to a schedule or testing program, as disclosed herein.Support center 124 may be connected to database 126 for storing trainingmaterial, interactions between user devices 102 and agent's devices 120,and/or interactions between validation bot 110 and agent's devices 120,and any other relevant data.

Support center 124 may include or may be connected to a validation bot110, which may send training material to agents and may coordinatelytest the coaching effectiveness by automatically initiating validationsessions, also referred to herein as validation interactions or calls,with the agents as disclosed herein. Validation bot 110, also referredto herein as validation process, may be a computer process that for thepurpose of testing the coaching effectiveness, may act as or pretend tobe a real customer, with which the agent is supposed to interact.Validation bot 110 and agent device 120 may be connected via one or moreof a web connection, a telephone connection, a messaging connectionand/or via one or more communication networks 140. Interactions orsessions between validation bot 110 and agent device 120 may use forexample voice/voice over internet protocol (VoIP), chat, texting ormessaging, micro-blogging, video, Internet, email and/or othercommunication channels.

System 100 may be connected, or configured to be connected, to one ormore user devices 102, such as, computers (for web or Internetconnections), telephones (for telephone or radio network connections),or messaging or text enabled devices (for messaging networkconnections), for a customer to interact with and communicate withagents over one or more networks 140. Customers may use user devices 102to connect to live or automated agents, for example, using or operatingon agent devices 120 at support center 124.

Reference is made to FIG. 2, which schematically illustrates a top-leveldesign of validation bot 110, according to embodiments of the invention.It should be readily understood that the structure of validation bot 110depicted in FIG. 2 is demonstrative only and the functionality disclosedherein may be performed by other modules or structures. Validation bot110 may include a coaching application 210, a bot scheduler 220, a botdesigner 230, a validation bot backend application 240, a customer mediainterface 250 a natural language understanding engine 260 and bot flow270. The various bot modules, and the bot itself, may be for exampleprocesses executed for example by a system as shown in FIG. 8.

Coaching application 210 may be configured to send coaching packages ortraining material to call center agents, e.g., via agent devices 120.Bot scheduler 220 may schedule validation sessions according to thecoaching plan configuration and may include a dispatcher 222 that maytrigger validation bot back end service 240 to start a session with atested agent according to the schedule. The session may be for example asurprise session or a known session. For example, a coaching planconfiguration may define when to send the coaching material to an agentor a group of agents, how much time should be given for the agents tolearn the material, and when should validation bot back end service 240be triggered to start validation sessions with the agents. In someembodiments bot scheduler 220 may obtain parameters such as for exampletraining plan identifier and name, list of tested agents, validation botidentifier, bot passing score and validation bot start and end dates.Other parameters may be used. In order to minimize the effect of thevalidation calls on the calling center, it may be desirable todistribute validation sessions across time. Additionally, validationsessions may be distributed such that the availability of agents to realcalls will not be affected, or be minimally affected. In someembodiments, bot scheduler 220 may be integrated or coordinated with awork force management (WFM) application, thus it may obtain informationregarding agent availability and may schedule the initiation ofvalidation sessions at times when agents are free. In some embodiments,there may be no integration with WFM application and bot scheduler 220may not have information regarding agent availability. In this case, botscheduler 220 may attempt to distribute validation sessions over time,so that not all or many agents will receive a validation session at thesame time, which might decrease the capacity of many agents to servecustomers at the same time. For example, bot scheduler 220 may attemptto distribute validation sessions as evenly as possible over time.

Bot designer application 230 may be an application configured to design,or to aid in the design of a validation bot or simulator based on thetraining material. Specifically, bot designer application 230 may beconfigured to design, or to aid in the design of bot flow 270. Botdesigner application 230 may obtain the agent intents, bot responseprompts or phrases, scoring rules, response rules, event rules and statevariables as inputs and may generate bot flow 270 based on these inputs.The validation bot may include a predetermined session flow or flows,also referred to herein as bot flow 270, that is related to or based onthe training material, e.g., the bot flow may be designed based on thetraining material in order to test that agents operate according to thetraining material. Bot flow 270 may include predefined agent intents,bot response prompts or phrases, scoring rules, response rules, eventrules and state variables. As used herein a phrase may relate to or bean expression, e.g., a word or a group of words conveying a certainmeaning. A bot response phrase or bot response may refer to a phrasethat a validation bot may provide to the agent. As used herein, anintent may refer to an intention or a goal of the agent. Exemplaryintents may include “greetings”, “sell car insurance”, etc. For example,phrases as “Do you also need car insurance?”, “Can I offer you our carinsurance policies?”, “Would you be interested in adding carinsurance?”, may all be mapped to or categorized as the intent “sell carinsurance”, while phrases as: “Hello”, “Hi”, “Good morning”, “How can Ihelp you?” may be mapped to or categorized as the intent “greetings”.

Bot response prompts may include sets of one or more response prompts orphrases that the bot may use to respond to an intent and a possibly to astate. For example, a set of prompts related to an “up sale carinsurance” intent may include: “Yes”, “What are your car insuranceofferings?”, “That could be interesting”, etc. Thus, in response to an“up sale car insurance” intent, bot back end service 240 may select onephrase from the set. The actual prompt may be chosen randomly. In someembodiments, if a specific intent is expected to be provided by theagent more than once in a single session, then bot flow 270 may considerthe current state in selecting a response from the relevant set. Forexample, a “Greetings” set of phrases for the “Greetings” intent mayinclude a plurality of phrases, intended to provide a natural feel forthe beginning of the interaction, but a “Greeting State” state variablemay remember the greetings phrases that were used so the bot back endservice 240 may not repeat itself and cause the interaction to progressbeyond the greetings phase. For example, bot flow 270 may include afirst “Greetings” set of phrases from which phrases may be randomlychosen with no repeat process, and after the first set is exhausted, asecond “Greetings” set including more aggressive phrases may be used toadvance the interaction. For example, the second set may include phrasessuch as: “I've called to renew my insurance, please could you provide anoffer”.

The bot state variables may include variables that may be used to definea state of bot back end service 240 in the interaction. The statevariables may be used as memory for remembering past actions of bot backend service 240, for example, how many and which phrases were usedbefore. For example, for the “Greetings” intent, bot flow 270 mayinclude a first “Greeting” state variable that may define that bot backend service 240 may provide up to three phrases from a first “Greetings”set and a second “Greeting” state variable that may define that bot backend service 240 may provide up to two phrases from a second “Greetings”set. During an interaction, the first “Greeting” state variable mayrecord or count the number of grating phrases provided by bot back endservice 240, and may be used to advance bot back end service 240 toproviding phrases from the second “Greetings” set if three phrases wereprovided from the first “Greetings” set and another “Greetings” wasaccepted from the agent. State variables may also be used forcalculating the score or rating of the agent. It should be readilyunderstood that other methods may be used for providing thefunctionality of the state variables.

The bot response rules may include rules that may define how to choose aresponse for a given intent and state. For example, a rule may be:

for a “Greeting” intent,

-   -   if the first “Greetings” set was used less than three times,        then randomly chose a response from the first “Greetings” set        and remove the chosen response from the first “Greetings” set,        else    -   if the second “Greetings” set was used less than two times, then        randomly chose a response from the second “Greetings” set and        remove the chosen response from the second “Greetings” set, else    -   set a score variable to zero and chose a response from the        “Failure” response set.

The bot scoring rules may include rules that may define the change tothe interaction score according to an intent and state. For example, the“up sale car insurance” intent may increase the score by 10 points, butonly in the first time it is recognized.

The event rules may include rules that may define a change to the statevariables according to events. For example, if the agent doesn't respondto a specific input within 10 minutes in a session, the score may bereduced by 10 points.

The termination or exit rules may include rules that define if that asession should end or if exit criterion has occurred. The terminationrules may also include a scoring formula that may be used forcalculating the total interaction score or rating based, for example, onvalues of the state variables of the interaction. The termination rulesmay also and pass\fail rules that may define whether the interaction issuccessful or not. For example, a scoring formula may be the accumulatedscore of all the defined score variables and a pass\fail rule may bethat if the accumulated score is above 70 the interaction is successful(e.g., pass) otherwise it is a failed test.

Validation bot backend application 240 may be responsible fororchestrating the flow of the bot conversation according to bot flow 270and as scheduled by bot scheduler 220, and may include the logic thatdetermines how the bot behaves.

Validation bot backend application 240 may receive a reply or answerfrom the agent. If the channel is voice, then the reply or inputs may betranscribed to text by an automatic speech recognition (ASR) engine. Thereply may be interpreted by natural language understanding engine 260,that may map or categorize the agent reply or responses into intents. Insome embodiments, natural language understanding engine 260 may includea classifier or other machine learning module that may be trained toclassify phrases into intents. In some embodiments natural languageunderstanding engine 260 may be or may include Microsoft's® languageunderstanding intelligent service (LUIS)®. In some embodiments, naturallanguage understanding engine 260 may return a list of possible intentsand their corresponding probability scores, also referred to asconfidence scores. The intent probability score may define or indicatethe level in which natural language understanding engine 260 isconfident that the phrase matches each intent. As part of the bot designpossible intents and possible set of phrases that are mapped to thoseintents may be provided to natural language understanding engine 260.Natural language understanding engine 260 may use these intents andphrases to infer the intents from the phrases together with its ownpre-trained language model. In some embodiments an “unidentified intent”may be defined, that may be returned in case the agent answer didn't fitany of the pre-defined intent.

Validation bot backend application 240 may obtain the intent, and basedon the intent, may update a state of the validation bot, provide aresponse to the agent according to a predetermined session flow, andupdate a score of the agent according to scoring rules.

Validation bot backend application 240 may conduct a session with thecoached agent via customer media interface 250. Customer media interface250 may translate the response into the media channel of the session,e.g., chat box, SMS, voice or other channels. If the channel is voicethen customer media interface 250 may use a voice synthesizer togenerate human like voice from the text in the session flow.

Validation bot backend application 240 may obtain details of the testedagent, the channel and a session flow. The details of the tested agentsmay be obtained from an automated call distribution (ACD) system and mayinclude a list of currently active agents, agent ID, channel type (e.g.,voice call, chat, SMS etc.), ACD queues, etc. Validation bot backendapplication 240 may initiate the session with the tested agent using therelevant channel via customer media interface 250 by calling the agent,either by chat, SMS, by voice or by other media channels. Once thesession is setup, validation bot backend application 240 may communicatewith the agent according to the session flow via customer mediainterface 250. Upon receiving answers from the agent, a dialog ofquestion-answer begins between validation bot 110 and the agent untilthe session or interaction is terminated. A session may be terminated bythe agent, or by validation bot backend application 240. For example,validation bot backend application 240 may close the session if atermination criterion or conditions indicating the process should bestopped is met.

Reference is now made to FIG. 3 which is a high-level flowchart of amethod for automatically testing coaching effectiveness by a validationbot, according to embodiments of the invention. In some embodiments, themethod for testing an agent may be performed by validation bot 110depicted in FIGS. 1 and 2. In operation 306 training material may besent to one or more agents, e.g., by coaching application 210. Inoperation 308 a session with a selected agent may be scheduled, e.g., bybot scheduler 220. In operation 310 a session with a selected agent maybe initiated, e.g., by dispatcher 222. The session may be for example avoice session or a written session over any applicable communicationchannel or standard, e.g., voice, voice over internet protocol (VoIP),chat, texting or messaging, micro-blogging, video, Internet, emailand/or other communication channels. The session may be initiated bycalling the agent and providing an opening phrase in accordance with botflow 270. In operation 312 a reply may be received from the agent. Insome embodiments, if no reply was received in a predefined timeoutperiod, the flow might end, or the bot could initiate another response.

In operation 314 it may be checked whether a termination criterion ismet or an exit term was reached. If a termination criterion is met, theflow may move to operation 332. Termination criterions, or reasons tostop the process may include, for example:

-   -   No reply was received in the defined timeout period.    -   Agent exceeds maximum allowed number of answers—bot flow 270 may        have a limitation on the number of allowed agent answers in a        single call. Exceeding this number may cause bot flow 270 to        terminate the call.    -   Agent failed a critical question—bot flow 270 may include        questions which may be defined as “critical questions”, meaning        that a failure by the agent to answer correctly these questions        may lead bot flow 270 to mark the session as failure regardless        of the agent answers in other questions.

If no termination criterion is met the flow may move to operation 316.In operation 316 the intent of the agent may be identified, by mapping,correlating or classifying the reply of the agent to an agent intent,e.g., using natural language understanding engine 260. In someembodiments, a list of possible intents and corresponding probabilityscores may be generated or calculated. In operation 318 the state of thevalidation bot may be updated based on the agent intent and on and rulesdefined by bot flow 270. The rules may include definitions as to how tochoose which intent to use based on the confidence score of the intents.In operation 320, a ranking or score of the agent may be calculated orupdated based on the agent intent and on scoring rules. For example, botflow 270 may include a list of intents that are expected to be expressedby the agent during the interaction. If an expected intent is receivedfrom the agent, the score may be increased according to a predeterminedformula. The formula for scoring may include weights for intents andother logic. In some embodiments, the score may be decreased if a numberof replies was exceeded. For example, for every agent answer startingfrom the 10^(th) answer the score may be decreased by 5 points.

In operation 322 it may be checked whether a termination criterion thatis based on the agent intent or updated score was met. For example, itmay be checked in this point if the agent has failed to answer a“critical question”, or whether the updated score is below apredetermined threshold. If no termination criterion was reached or met,then in operation 324 a response to the agent may be provided. Theresponse may be selected based on the agent intent and according to apredetermined session flow. The response may be based on a state ofvalidation bot 110. The response may be provided via customer mediainterface 250, in the appropriate communication channel and standard. Ifthe intent is not identified, the bot may return some generic responseor a request for clarifications.

Reference is now made to FIG. 4 which is a high-level flowchart ofoperations that may be performed in case of a failed test, according toembodiments of the invention. A failed test in a validation interactionmay be where the agent has failed the verification test (e.g., asdetermined in operation 330 in FIG. 3). In some embodiments, theseoperations may be performed by validation bot 110 depicted in FIGS. 1and 2. After a test has ended, as indicated in block 410, then one ormore of the following may be performed automatically:

-   -   If an agent has failed the validation test, training material        may be sent to the agent, as indicated in block 420. In some        embodiments, the same training material that was sent in        operation 306 may be resent again. In other embodiments other        training material, e.g., more detailed training material, may be        sent.    -   pop-ups, e.g., text provided on a screen of agent devices 120,        with correct answers as may be defined in bot flow 270, may be        provided to the agent in real-time, as indicated in block 430        this may be performed during the validation test as well as        after the test has ended.    -   the agent may be informed of his score, as indicated in block        440.    -   If a large group of agents have failed, or if a percentage of        agents that have passed the verification test is below a        predetermined threshold, an indication that the training        material should be refined or improved may be provided, as        indicated in block 450. In some embodiments, validation bot 110        may provide a detailed report showing which questions were not        answered properly by how many agents, thus providing a more        specific insight as to what parts of the training material        should be improved.    -   A coaching plan of a specific agent or a group of agents may be        updated according to the validation test results, as indicated        in block 460.

Reference is made to FIG. 5, which schematically illustrates a partialoperational flow of validation bot backend application 240, according toembodiments of the invention. A logic engine 540 of validation botbackend application 240 may obtain intents with scores 510, e.g., fromnatural language understanding engine 260 and a current state 520, andmay calculate a new state 560 based on intents with scores 510 andcurrent state 520. New state 560 may be defined by updated values of thebot state variables and the bot response phrase that may be provided tothe agent. Logic engine 540 may also calculate an updated agent scorebased on intents with scores 510 and current state 520.

The bot may also change its state according to the time it takes theagent to reply. For example, if the agent does not respond to a specificinput within 10 minutes in chat, the score may be reduced by 10 pointsand the bot will tell the agent that it takes him too long to respond asa real customer might behave. In operation 326 the validation bot maywait for input or reply from the agent. Once a reply is received, thevalidation bot may return to operation 312.

Returning to FIG. 3, if a termination criterion is met, either inoperation 314 or in operation 322, then the session terminates, ends orstops in operation 332 and in operation 328 a final score may becalculated, and in operation 330 a Boolean pass/fail indication may beprovided. For example, it may be determined that the agent has passedthe verification test if the score of the agent exceeds a predeterminedthreshold and that the agent has failed the verification test if thescore does not exceed the threshold or if the agent failed a criticalquestion. The score and the pass/fail indication may be provided to auser or to the initiator of the validation e.g., coaching application210. The score and the pass/fail indication may be stored in a database,e.g., database 126. After providing the score and/or pass/failindication the bot may end the session.

As indicated herein, it may be desirable to distribute validationsessions over time. If bot scheduler 220 is not integrated orcoordinated with a WFM application then determining timing of initiatingsessions with agents that need to be validated may be performed based ona number of the agents that need to be validated and a time availablefor validating the agents. Timing may be determined so that the sessionsare evenly distributed over the time available for validating theagents. A very simple way to achieve that may be to divide the number ofagents that need to be validated by the time available for validatingthe agents to arrive at a time interval, and initiating a session withan agent that needs to be validated every time interval. If the agent isnot available, a session may be initiated with another agent that needsto be validated. Another method for evenly distributed sessions is givenbelow.

Reference is now made to FIG. 6 which is a high-level flowchart of amethod for evenly distributing sessions, according to embodiments of theinvention. In some embodiments, the method for evenly distributingsessions may be performed by bot scheduler 220 depicted in FIG. 2.According to some embodiments, scheduling of sessions for a relevantgroup of agents may be done over a period of time, and bot scheduler 220may aim to spread the initiation of the sessions as evenly as possibleover the time period. The motivation for evenly spreading verificationsessions is to minimize the chances that many agents will receive avalidation call at the same time and the availability of the callcenter, e.g., support center 124, may decrease. Embodiments of themethod presented in FIG. 6 may be used when there is no informationregarding agents shifts plan, staffing and expected loads, e.g., whenthere is no connection with a WFM application.

The following terminology may be used for describing the embodiment ofFIG. 6:

-   -   total_agents_to_validate—list of agents that needs to receive a        validation session.    -   agents_already_validated—list of agents for which a validation        session was completed.    -   agents_not_yet_validated—list of agents for which a validation        session was not yet launched.    -   date_to_complete_validation—the date by which all validation        sessions need to be completed.    -   time_slots_to_complete_validation—how many time slots, e.g.,        working hours (or other time slot duration) remains before        date_to_complete_validation date. This value may be calculated        based on the difference between current time and        date_to_complete_validation, taking into account working days        and working hours in each day.    -   active_agents—list of agents who are currently active in the        contact center.

In operation 602 the following parameters may be obtained:total_agents_to_validate, and date_to_complete_validation. In operation610 the time available for completing the validation may be calculated,for example, time_slots_to_complete_validation parameter may becalculated based on current time and date_to_complete_validationparameter. Operation 610 may occur periodically, e.g., everypredetermined time interval, while validation continues and duringworking hours, for example every hour. In operation 620 thenumber_of_sessions_to_dispatch_next_time_slot, e.g., the number ofagents that should receive a validation session in coming time slot maybe calculated, for example by dividing the number ofagents_not_yet_validated by time_slots_to_complete_validation. Forexample, the number of agents that should receive validation bot sessionbefore the completion date may be divided by the number of time slotsremaining until the end date. This calculation aims to evenly distributevalidation sessions across working hours.

In operation 630 the active_agents list may be scanned to find agentsthat need to be verified, e.g., agents that are included in theagents_not_yet_validated list. In some embodiments the list may beobtained from the ACD system. In operations 640, sessions may beinitiated with the agents that were found in operation 630, e.g., bydispatcher 222 depicted in FIG. 2. In case a session was initiated withthe agent, then in operation 650 the agents_not_yet_validated list maybe updated, e.g., the agent who received the validation session may beremoved from the agents_not_yet_validated list and added toagents_already_validated list. In operation 660 it may be determinedwhether the validations have ended. The validation may end if all of theagents were validated, or when date_to_complete_validation has arrived.If by the end of the period (date=date_to_complete_validation) there arestill agents who didn't receive the validation bot session(agents_not_yet_validated>0) then an alert may be provided to a user.The user may decide whether to extend the date_to_complete_validation orterminate the flow without all agents receiving validation bot sessions.If the validation has not ended then the flow may return to operation610. If the validation has ended, then in operation 670 validation maybe terminated. In some embodiments a list of agents_not_yet_validated,if any, may be provided to the user.

In some embodiments bot scheduler 220 may obtain information regardingagents shifts and staffing, e.g., by being integrated or coordinatedwith a WFM application. Thus, scheduling may be performed taking intoaccount agents shifts and staffing. For example, sessions may bescheduled for times in which there are more agents than an expectednumber required for answering human clients. In some embodiments,sessions may be scheduled for times in which a difference between anexpected number of agent required for answering human clients and anactual number of agents is above a predetermined threshold.

Reference is now made to FIG. 7 which is a high-level flowchart of amethod for distributing sessions according to agent's availability,according to embodiments of the invention. In some embodiments, themethod for distributing sessions according to agent's availability maybe performed by bot scheduler 220 depicted in FIG. 2. In someembodiments, agent's availability may be obtained from a WFMapplication. Agent's availability information may include agent'sshifts, e.g., which agents work at what times, staffing, e.g., how manyagents are working per time, and an expected number of agents requiredfor meeting the service level agreement (SLA, a commitment to answer acertain percentage of calls within a certain time) of the contactcenter. The expected number of agents required for meeting the contactcenter SLA may be referred to herein as the required number of agents.The required number of agents may be determined or estimated by aforecaster (not shown) at the contact center. Coordinating validationsessions with agent's availability may minimize the effect thatvalidation sessions may have on the contact center, the effect is due tooccupying the agents with bot training activity and not with answeringcustomers. Embodiments of the invention may aim at minimizing the effectof the validation sessions by trying to find times in which the effectof reducing the available agents for answering validation calls will beminimal.

The following terminology may be used for describing the embodiment ofFIG. 7:

-   -   validation_end_date—the date by which all validation sessions        need to be completed.    -   Time slot—the time from the beginning of the validation to the        validation_end_date may be divided into time slots, e.g., of        half an hour, one hour, two hours or other duration.    -   “over staffing” time slot—a time slot during contact center        working hours in which there are more agents than the required        number of agents.    -   “under staffing” time slot—a time slot during contact center        working hours in which there are less agents than required        number of agents.    -   agents_already_validated—list of agents for which a validation        session was completed.    -   agents_not_yet_validated—list of agents for which a validation        session was not yet launched.    -   staffing_threshold_for_bot_scehduling—a threshold on the        difference between the actual number of agents and the required        number of agents. For example, verification sessions may be        scheduled for time slots in which the difference between the        actual number of agents and the required number of agents is        equal to or above the staffing_threshold_for_bot_scehduling. The        staffing_threshold_for_bot_scehduling may have either positive        or negative value. For example, if        staffing_threshold_for_bot_scehduling equals three, then        sessions may be initiated only when there is overstaffing of        three agents or more, and If the        staffing_threshold_for_bot_scehduling is set to −2 (minus two),        then sessions may be initiated even when there is understaffing        of two agents or less.    -   agents_scheduled_for_bot—a list including details of the agents        scheduled for a validation session, including, for example, the        agent name, time of initiation and the specific bot flow to        initiate.    -   agents_not_scheduled_for_bot—list of agents for which the        scheduler couldn't find any time slot under the algorithm        constraints.    -   The agents that are in the agents_scheduled_for_bot and        agents_not_scheduled_for_bot together compose the list of agents        in agents_not_yet_validated list.    -   staffing_gap—the difference between the required number of        agents and the actual number of agents at the contact center at        a given time slot. A positive staffing_gap is equivalent to        “overstaffing”, and a negative staffing_gap is equivalent to        “understaffing”.

In operation 702 a list of all agents that needs to be validated,agents_not_scheduled_for_bot, may be obtained, e.g., from bot back endservice 240. At this stage agents_scheduled_for_bot list may be empty.In operation 704 agent's shift plan and staffing of the contact centerfrom current date until validation_end_date, as well as the requirednumber of agents at each time slot, may be obtained, e.g., from a WFMapplication. In operation 706 a staffing_gap may be calculated for eachtime slot as the difference between the required number of agents andthe actual staffing at each time slot. In operation 708, all time slotswith staffing_gap that is lower than thestaffing_threshold_for_bot_scehduling may be removed. In operation 710it may be verified if there are relevant time slots remaining, e.g., ifthere are still available time slots with staffing_gap that is equal toor higher than the staffing_threshold_for_bot_scehduling.

In operation 712 the time slot with the highest staffing_gap may beselected. For example, the remaining time slots may be sorted indescending order (the time slot with highest staffing_gap may be first)and the first time slot may be selected. The sorted time slots may bestored in an ordered list called “ordered_time_slot”. In operation 714,it may be determined if agent is in the agents_not_scheduled_for_botlist are scheduled to work at the contact center during the selectedtime slot, e.g., based on the agent's shifts plan. If no agents arescheduled to work at the contact center during the selected time slot,then in operation 716 the time slot may be removed from considerationsince not agents that need to be verified are working at the call centerduring this time slot, and the flow may return to operation 710. If inoperation 714, an agent from the agents_not_scheduled_for_bot list isfound to be scheduled to work at the contact center during the selectedtime slot, then in operation 718 the agent may be removed from theagents_not_scheduled_for_bot list, and added to theagents_scheduled_for_bot list, with the current time slot as anassociated “launch time” parameter. Also in operation 718, thestaffing_gap of the selected time slot may be decreased by one, since anagent was allocated to receive a verification session in this time slot,effectively reducing the number of agents available for handlingcustomers by 1.

In operation 720, it may be verified whetheragents_not_scheduled_for_bot list is empty, e.g., whether all agentsthat were originally in that list were scheduled for a verificationsession. If there are agents remaining in theagents_not_scheduled_for_bot list, then the flow may return to operation708, to remove time slots with staffing_gap below thestaffing_threshold_for_bot_scehduling, e.g., the time slot selected inoperation 712 may be removed if after allocating a verification sessionto an agent in this time slot, the staffing_gap of that time slot hasdecreased below the staffing_threshold_for_bot_scehduling.

If it is determined in operation 720 that theagents_not_scheduled_for_bot list is empty, or if there are no timeslots with staffing_gap equal or above thestaffing_threshold_for_bot_scehduling then scheduling is complete, theagents_scheduled_for_bot and agents_not_scheduled_for_bot lists arepopulated correctly, and the flow moves to operation 722. In someembodiments, operations 702-720 may be repeated periodically, or if achange in the agent's availability information has occurred.

In operation 722 validation sessions may be initiated according to theagents_scheduled_for_bot list. For example, the list ofagents_scheduled_for_bot may be scanned periodically, e.g., everyminute, every 10 minutes etc., to see whether a “launch time” parameterof sessions in the agents_scheduled_for_bot list has arrived, e.g., ifthe current time is higher than the “launch time” specified in theagents_scheduled_for_bot list. Operation 722 may be performed by adispatcher block 222 depicted in FIG. 2. If a session is initiated andperformed, the agent may be removed from the agents_scheduled_for_botlist and is moved to the agents_already_validated list. In case offailure in initiating the validation session, the agent may be removedfrom the agents_scheduled_for_bot list and moved to theagents_not_scheduled_for_bot. In this case operations 702-720 may berepeated to allocate a new time slot for a new validation bot sessionfor the agent. A coaching service or a user may be updated with allagents in the agents_not_scheduled_for_bot list.

Reference is made to FIG. 8, showing a high-level block diagram of anexemplary computing device according to some embodiments of the presentinvention. Computing device 800 may include a controller 805 that maybe, for example, a central processing unit processor (CPU), a graphicsprocessing unit (GPU), a chip or any suitable computing or computationaldevice, an operating system 815, a memory 820, executable code 825,storage or storage device 830, input devices 835 and output devices 840.Controller 805 may be configured to carry out methods described herein,and/or to execute or act as the various modules, units, etc., forexample by executing code or software. More than one computing device800 may be included. For example, by executing executable code 825stored in memory 820, controller 805 may be configured to carry out amethod for testing agents by a validation bot as described herein. Eachof user devices 102, support center 124 validation bot 110 and agentdevices 120 may be or may include computing device 800 or componentssimilar to device 800. Processes such as a validation bot 110 may beexecuted by a device similar to device 800.

Operating system 815 may be or may include any code segment (e.g., onesimilar to executable code 825 described herein) designed and/orconfigured to perform tasks involving coordination, scheduling,arbitration, supervising, controlling or otherwise managing operation ofcomputing device 800, for example, scheduling execution of softwareprograms or enabling software programs or other modules or units tocommunicate. Operating system 815 may be a commercial operating system.

Memory 820 may be or may include, for example, a Random Access Memory(RAM), a read only memory (ROM), a Dynamic RAM (DRAM), a SynchronousDRAM (SD-RAM), a double data rate (DDR) memory chip, a Flash memory, avolatile memory, a non-volatile memory, a cache memory, a buffer, ashort term memory unit, a long term memory unit, or other suitablememory units or storage units. Memory 820 may be or may include aplurality of, possibly different memory units. Memory 820 may be acomputer or processor non-transitory readable medium, or a computernon-transitory storage medium, e.g., a RAM.

Executable code 825 may be any executable code, e.g., an application, aprogram, a process, task or script. Executable code 825 may be executedby controller 805 possibly under control of operating system 815. Forexample, executable code 825 may be an application that when executedgenerates pseudo-realistic data as further described herein. Although,for the sake of clarity, a single item of executable code 825 is shownin FIG. 8, a system according to embodiments of the invention mayinclude a plurality of executable code segments similar to executablecode 825 that may be loaded into memory 820 and cause controller 805 tocarry out methods described herein. For example, units or modulesdescribed herein may be, or may include, controller 805 and executablecode 825.

Storage device 830 may be any applicable storage system, e.g., a disk ora virtual disk used by a VM. Storage 830 may be or may include, forexample, a hard disk drive, a floppy disk drive, a Compact Disk (CD)drive, a CD-Recordable (CD-R) drive, a Blu-ray disk (BD), a universalserial bus (USB) device or other suitable removable and/or fixed storageunit. Content or data may be stored in storage 830 and may be loadedfrom storage 830 into memory 820 where it may be processed by controller805. In some embodiments, storage device 830 may be used for storingdata related to generating pseudo-realistic data. In some embodiments,some of the components shown in FIG. 8 may be omitted. For example,memory 820 may be a non-volatile memory having the storage capacity ofstorage 830. Accordingly, although shown as a separate component,storage 830 may be embedded or included in memory 820. Database 126 maybe implemented over storage 830.

Input devices 835 may be or may include a mouse, a keyboard, a touchscreen or pad or any suitable input device. It will be recognized thatany suitable number of input devices may be operatively connected tocomputing device 800 as shown by block 835. Output devices 840 mayinclude one or more displays or monitors, speakers and/or any othersuitable output devices. It will be recognized that any suitable numberof output devices may be operatively connected to computing device 800as shown by block 840. Any applicable input/output (I/O) devices may beconnected to computing device 800 as shown by input devices 835 andoutput devices 840. For example, a wired or wireless network interfacecard (NIC), a printer, a universal serial bus (USB) device or externalhard drive may be included in input devices 835 and/or output devices840.

Some embodiments of the invention may include an article such as acomputer or processor non-transitory readable medium, or a computer orprocessor non-transitory storage medium, such as for example a memory, adisk drive, or a USB flash memory, encoding, including or storinginstructions, e.g., computer-executable instructions, which, whenexecuted by a processor or controller, carry out methods disclosedherein. For example, an article may include a storage medium such asmemory 820, computer-executable instructions such as executable code 825and a controller such as controller 805.

The storage medium may include, but is not limited to, any type of diskincluding, semiconductor devices such as read-only memories (ROMs)and/or random access memories (RAMs), flash memories, electricallyerasable programmable read-only memories (EEPROMs) or any type of mediasuitable for storing electronic instructions, including programmablestorage devices. For example, in some embodiments, memory 820 is anon-transitory machine-readable medium.

A system according to some embodiments of the invention may includecomponents such as, but not limited to, a plurality of centralprocessing units (CPU) or any other suitable multi-purpose or specificprocessors or controllers (e.g., controllers similar to controller 805),a plurality of input units, a plurality of output units, a plurality ofmemory units, and a plurality of storage units. A system according tosome embodiments of the invention may additionally include othersuitable hardware components and/or software components. In someembodiments, a system may include or may be, for example, a personalcomputer, a desktop computer, a laptop computer, a workstation, a servercomputer, a network device, or any other suitable computing device. Forexample, a system according to some embodiments of the invention asdescribed herein may include one or more devices such as computingdevice 800.

Different embodiments are disclosed herein. Features of certainembodiments may be combined with features of other embodiments; thus,certain embodiments may be combinations of features of multipleembodiments.

Embodiments of the invention may include an article such as a computeror processor readable non-transitory storage medium, such as for examplea memory, a disk drive, or a USB flash memory device encoding, includingor storing instructions, e.g., computer-executable instructions, whichwhen executed by a processor or controller, cause the processor orcontroller to carry out methods disclosed herein.

While the invention has been described with respect to a limited numberof embodiments, these should not be construed as limitations on thescope of the invention, but rather as exemplifications of some of thepreferred embodiments. Other possible variations, modifications, andapplications are also within the scope of the invention. Differentembodiments are disclosed herein. Features of certain embodiments may becombined with features of other embodiments; thus, certain embodimentsmay be combinations of features of multiple embodiments.

What is claimed is:
 1. A computer implemented method for testing anagent by a validation bot executed by a processor, the methodcomprising: a. sending training material, by the validation bot, to anagent; b. at the same time that agents other than the agent areconnected to customers, automatically initiating, by the validation bot,a session with an agent by providing a natural language phrase to theagent via a client media interface, wherein the client media interfaceis the same interface the agent uses to communicate with a customer,wherein, during the session, the validation bot pretends to be a realcustomer with which the agent is supposed to interact; c. obtaining, bythe validation bot, a reply from the agent; d. interpreting the reply bya natural language engine which includes a machine learning moduletrained to classify agent replies into agent intents and mapping, by thevalidation bot, the reply of the agent to an agent intent, wherein theagent intent is a goal of the agent expressed by the agent during thesession; and e. providing, by the validation bot, a response to theagent based on the mapped agent intent.
 2. The method of claim 1,wherein the providing of the response is according to the mapped agentintent and on a predetermined session flow, wherein the predeterminedsession flow is based on the training material, wherein the response istranslated into a media channel of the session, and wherein thepredetermined session flow comprises possible agent intents, possibleresponses to the agent, scoring rules, response rules, and statevariables, wherein a state of the validation bot is updated based on theagent intent, and wherein providing the response to the agent isperformed based on the state of the validation bot and on the botresponse rules.
 3. The method of claim 1, comprising calculating, by thevalidation bot, a score of the agent according to scoring rules.
 4. Themethod of claim 1, comprising repeating operations a-e for a pluralityof agents.
 5. The method of claim 4, comprising: determining timing ofinitiating sessions with agents that need to be validated based on anumber of the agents that need to be validated and a time available forvalidating the agents, wherein timing is determined so that the sessionsare evenly distributed along the time available for validating theagents.
 6. The method of claim 1, comprising scheduling the sessions fortimes in which there are more agents than an expected number requiredfor answering human clients.
 7. The method of claim 1, comprisingscheduling the sessions for times in which a difference between anexpected number of agents required for answering human clients and anactual number of agents is above a predetermined threshold.
 8. Themethod of claim 3, comprising providing a recommendation to refinetraining material if a percentage of agents that have passed averification test based on the score of the agent is below apredetermined threshold.
 9. The computer implemented method of claim 1wherein the client media interface is a chat box, a short messageservice (SMS), voice, or any combination thereof.
 10. The method ofclaim 1 comprising repeating, by the validation bot, operations c-euntil a termination criterion is met.
 11. A computer implemented methodfor simulating a human client for testing agents, the method comprising:a. initiating, by an automatic validation process executed by aprocessor a session with the agent by providing a natural languagephrase to the agent via a client media interface, wherein the clientmedia interface is the same interface the agent uses to communicate witha customer, the initiating occurring at the same time that agents otherthan the agent are connected to customers, wherein, during the session,the processor pretends to be a real customer with which the agent issupposed to interact; b. receiving an agent reply from the agent,wherein the agent intent is a goal of the agent expressed by the agentduring the interaction; c. correlating the agent reply with an agentintent using a natural language understanding engine which includes amachine learning module trained to classify agent replies into agentintents; d. updating a state of the validation process based on theagent intent; e. providing a response to the agent based on the agentintent; f. calculating a rating of the agent according to scoring rules;and g. providing the rating of the agent to a user.
 12. The method ofclaim 11, comprising providing a response to the agent based on theagent intent and on the state of the validation process, wherein theresponse is translated into a media channel of the interaction.
 13. Asystem for testing an agent by a validation bot, the system comprising:a memory; a processor configured to: a. send training material to anagent; b. automatically initiate a session in which the validation botpretends to be a real human customer interacting with an agent, at thesame time that agents other than the agent are connected to customers,by providing a phrase to the agent via a client media interface, whereinthe client media interface is the same interface the agent uses tocommunicate with a customer; c. obtain a reply from the agent; d.interpret the reply by a natural language engine which includes amachine learning module trained to classify agent replies into agentintents and map the reply of the agent to an agent intent, wherein theagent intent is a goal of the agent expressed by the agent during thesession; and e. provide a response to the agent based on the agentintent.
 14. The system of claim 13, comprising providing the responsebased on the agent intent and according to a predetermined session flow,wherein the predetermined session flow is based on the trainingmaterial, wherein the response is translated into a media channel of thesession, and wherein the predetermined session flow comprises possibleagent intents, possible responses to the agent, scoring rules, responserules, and state variables, wherein a state of the validation bot isupdated based on the agent intent, and wherein providing the response tothe agent is performed based on the state of the validation bot and onthe bot response rules.
 15. The system of claim 13, comprising repeatingoperations a-e for a plurality of agents.
 16. The system of claim 13,wherein the processor is configured to determine timing of initiatingsessions with agents that need to be validated based on a number of theagents that need to be validated and a time available for validating theagents, wherein timing is determined so that the sessions are evenlydistributed along the time available for validating the agents.
 17. Thesystem of claim 13, wherein the processor is configured to schedule thesessions for times in which there are more agents than an expectednumber required for answering human clients.
 18. The system of claim 13,wherein the processor is configured to schedule the sessions for timesin which a difference between an expected number of agent required foranswering human clients and an actual number of agents is above apredetermined threshold.
 19. The system of claim 13, wherein theprocessor is configured to calculate, by the validation bot, a score ofthe agent according to scoring rules.
 20. The system of claim 19,comprising if the score is below a predetermined threshold performingone of: sending training material to the agent, providing pop-ups withcorrect answers to the agent in real-time, and informing the agent ofthe low score.